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Objectives 


•  Independently  review  cunent  and  planned  prefligbt  testing  practices 
for  hit>to-kiU  (HTK)  ballistic  missile  defense  (BMD)  interceptor 
programs,  assess  their  adequacy,  and  identify  any  innovations  that 
might  be  needed  to  provide  a  high  level  of  confidence  that  each  flight 
test  will  be  successful 

•  Identify  best  practices  for  the  National  Missile  Defense  (NMD) 
program  vs.  specific  program  reconunendations 

•  Sponsors 

-  Director,  Operational  Test  and  Evaluation  (DOT&E) 

-  Director,  Test,  Systems  Engineering  and  Evaluation  (DTSE&E) 

-  Director,  Ballistic  Missile  Defense  Organization  (BMEX)) 


This  study  was  initiated  by  the  sponsors  to  address  risk  in  the  flight  test 
programs  of  BMDO’s  hit-to-kill  (HTK)  ballistic  missile  defense  (BMD) 
systems.  The  four  systems  are  the  Theater  High  Altitude  Area  Defense 
(THAAD);  the  Patriot-3  System  with  its  Patriot  Advanced  Capability-3 
(PAC-3)  missile;  the  AEGIS  LEAP  Interceptor  (ALI)  Program,  which  is  a 
risk-reduction  program  within  the  Navy  Theater  Wide  (NTW)  Defense 
System;  and  BMDO’s  NMD  program.  This  study  was  motivated  by  a  series  of 
flight  test  failures  in  some  of  these  programs— failures  which  indicated  a  high 
level  of  risk.  These  failures  have  significantly  delayed  the  planned  fielding  of 
BMD  systems. 

The  study  group  members  examined  current  theater  missile  defense 
(TMD)  programs  to  address  the  issues.  However,  the  task  was  not  to  evaluate 
these  programs.  Rather,  the  task  was  to  take  lessons  from  these  programs  that 
could  and  should  be  applied  to  the  NMD  program. 

Observations  about  the  current  state  and  future  progress  on  these 
individual  TMD  programs  are  relevant  to  the  findings  of  the  study  and  are 
included  here.  We  focused  on  those  observations  that  are  common  to  more 
than  one  program  and  that  could,  therefore,  be  important  warning  flags  for  the 
NMD  program. 
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study  Group  Members 


•  Gen  Lany  Welch,  USAF  (ReL),  Task  Uader 

•  Mr.  Charles  "Pete”  Adolph 

•  Dr.  Pamey  Albright 

•  LtGcn  Aloysius  Casey,  USAF  (Ret) 

•  Dr.  Charlie  Cook 

•  Mr.  Edgar  Coitright 

•  MG  Eugene  Fox.  USA  (Ret.) 

•  Mr.  Mike  Fossier 

•  Mr.  Gerald  Johnston 

•  Dr.  Paul  Kaminski 


Mr.  Frank  Kendall 
Dr.  Steve  Kramer 
LTG  Don  Lionetti,  USA  (Ret.) 
RADM  Wayne  Meyer.  USN  (Ret.) 
Mr.  Robert  Pedraglia 
Dr.  Maile  Smith 
Government  Advisors 
"  Mr.  Craig  Farr 
“  LtCol  Dallas  Ferneau.  USAF 
-  Mr.  Larry  Miller 


The  study  group  was  composed  of  members  who  have  extensive 
experience  in  the  development,  testing,  and  operational  employment  of 
complex  systems. 

We  applied  this  experience  in  examining  current  HTK  BMD  programs. 
The  goal  was  to  extract  lessons  from  these  programs  and  other  complex 
programs  and  apply  these  lessons  to  the  NMD  program. 
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Tasks 

•  Review  the  historic  test  and  evaluation  (T&E)  paradigms  for  testing  missile 
systems  to  see  if  these  paradigms  are  still  relevant,  practical,  and  affordable 

•  Review  past  and  present  programs  to  identify  factors  that 

-  Contribute  to  maximizing  the  amount  learned  from  each  flight  test 

-  Enhance  the  success  of  each  flight 

•  Examine  how  best  to  identify  the  most  likely  failure  modes  and  the  key  factors 
in  the  success  (or  lack  thereoO  of  interceptor  flight  tests 

•  Recommend  practices  and  preflight  test  options  that,  if  implemented  by 
BMDO,  would  mitigate  the  risk  of  flight  test  failures 

•  Address  how  the  NMD  Joint  Program  Office  (JPO)  could  best  use  additional 
resources  allocated  in  the  Quadrennial  Defense  Review  (QDR) 


These  tasks  further  defined  the  overall  objective. 

As  previously  indicated,  to  understand  the  historic  T&E  paradigm,  the 
study  group  relied  extensively  on  members’  broad  and  extensive  experience  in 
dealing  with  complex  development  programs.  The  primary  focus  was  on 
reviewing  current  relevant  programs,  relating  the  results  of  this  review  to 
history  and  experience,  and  applying  the  lessons  to  the  NMD  program. 

We  found  that  the  processes  and  approaches  to  identify  and  preclude 
failure  modes  in  BMD  programs  are  not  fundamentally  different  from  the  long¬ 
standing,  sound  design  and  management  approaches  used  for  other  successful 
systems.  However,  the  demands  of  the  HTK  end  game  require  a  higher  level 
of  Hdelity  in  the  ground  simulations  [both  digital  and  hardware-in-the-loop 
(HWIL)]  to  surround  the  variables  and  uncertainties  of  the  dynamics  and  target 
presentation. 

This  report  presents  specific  findings  on  practices  and  preflight  test  options 
and  on  NMD  program  areas  that  require  increased  funding  emphasis. 
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Report  Outline 

•  Objectives,  Sponsors,  and  Study  Group  Members 

•  Tasks 

•  Key  Judgments — Overarching  Observations 

•  Recommendations  for  Reducing  Flight  Test  Risk  and  Fielding  Capability 
Expeditiously 

-  All  BMD  programs 

-  NMD 

•  Findings:  Responses  to  SpeciHc  Questions  in  the  Tasking 

•  Specific  Program  Reviews 

-  Peacekeeper  ICBM  Program 

-  THAAD 

-  NTW  Theater  Ballistic  Missile  Defense  (TBMD) 

-  Patriot  Missile  Program 

-  The  NMD  Program 

•  Meetings  and  Site  Visits 

•  Glossary 


In  keeping  with  the  study  group  objectives,  the  Key  Judgments  are 
overarching  study  group  views  based  on  the  aggregate  of  HTK  programs 
examined. 

Specific  recommendations  are  provided  for  reducing  the  test  risk  in  the 
NMD  program.  Findings  that  address  the  specific  questions  in  the  tasking  are 
also  provided. 

In  addition,  we  reviewed  practices  from  other  complex  programs.  In  this 
category,  even  though  the  report  specifically  discusses  only  the  Peacekeeper 
Intercontinental  Ballistic  Missile  OCBM)  program  and  the  earlier  Sprint 
Missile  program,  our  experiences  with  other  programs  provide  similar 
conclusions. 

The  sections  on  specific  BMD  programs  highlight  some  of  the  sources  of 
the  lessons  that  we  believe  are  particularly  relevant  to  the  NMD  program. 
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Key  Judgments — 
Overarching  Observations 


This  section  begins  with  the  Key  Judgments.  These  Key  Judgments 
emanated  from  what  the  study  group  regarded  as  the  most  overarching  in  the 
aggregate  of  the  programs  examined  and  the  experiences  of  the  study  group 
members.  These  are  also  those  most  important  as  lessons  for  NMD. 

Following  the  Key  Judgments,  overarching  recommendations  arc  provided 
and  then  specific  questions  are  addressed — ^again  using  the  lessons  from  the 
programs  examined  and  experiences  of  the  study  group  members. 


Key  Judgments  (1  of  6) 

The  Program  Management  Environment 

•  HTK  remains  a  difficult  technical  challenge 

-  Prudent  management  demands  cardul  attention  to  avoid  increasing  the  inherent  risk 
further 

•  Failures  to  date  have  little  to  do  with  HTK  vehicle  technology  but  have 
prevented  the  demonstration  of  that  technology 

«  The  flight  test  record  to  date  has  not  demonstrated  that  the  technology  has 
reached  the  state  of  maturity  needed  for  operational  systems 

-  Homing  Overlay  Experiment  (HOE):  1  intercept  in  4  attempts 

-  THAAD:  4  intercept  attempts,  4  failures 

-  Lightweight  Exo-Atmospheric  Projectile  (LEAP):  4  intercept  attempts, 

4  failures 

-  Extended  Range  Interceptor  (ERINT):  2  intercepts  in  3  attempts  at  theater 
ballistic  missile  (TBM)  targets 

-  Exo-atmospheric  Reentry  Vehicle  Intercept  System  (ERIS):  1  intercept  in 
2  attempts 


The  study  group’s  most  fundamental  finding  and  the  finding  most  relevant 
to  the  NMD  program  is  that  the  general  planning  and  execution  of  the  THAAD 
and  LEAP  programs  are  inconsistent  wiA  the  difficulty  of  the  task.  These 
programs  are  pursuing  very  aggressive  schedules,  but  these  schedules  are  not 
supported  by  the  state  of  planning  and  testing. 

Specifically,  the  perceived  urgency  of  the  need  for  these  systems  has  led  to 
high  levels  of  risk  that  have  resulted  in  delayed  deployments  because  of 
failures  in  their  development  test  (DT)  programs. 

After  more  than  a  dozen  flight  tests,  the  most  obvious  and  visible 
consequence  of  this  approach  is  that  we  are  still  on  “step  one”  in  demonstrating 
and  validating  HTK  systems.  Failures  having  little  to  do  with  the  kill  vehicle 
(KV)  performance — ^where  the  technology  should  be  in  hand — ^have  precluded 
demonstrating  that  weapons  systems  are  capable  of  reliably  hitting  a  ballistic 
missile  warhead.  And  even  when  this  first  step  is  achieved,  these  programs 
will  still  have  to  go  through  steps  two  and  three:  demonstrating  reliable  HTK 
at  a  weapon  system  level  and  demonstrating  reliable  HTK  against  likely  real- 
world  targets. 

Failures  to  date  reflect  inadequate  design  and  fabrication  discipline. 
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Key  Judgments  (2  of  6) 

The  Program  Management  Environment 

•  Programs  have  been  characterized  by  pressures  for  higher  risk  approaches  to 
meet  an  “urgent  need"  for  early  capability  [e.g..  THAAD  User  Operational 
Evaluation  System  (UOES)],  but  this  capability  is  inconsistent  with  the  technical 
chaUenge 

•  Program  “urgency”  is  reflected  in  less-than-minimal  or  highly  compressed 
planned  flight  testing 

-  THAAD:  20  flight  tests  in  24  months 

•  Now  13  flight  tests*  with  the  schedule  continuing  to  slip 

•  1  intercept  required  to  exercise  the  40>missile  UOES  buy 

-  NTW:  9  flight  tests  in  48  months 

-  Patriot  PAC-3:  16  tests  in  2+  years  (1 1  BMD) 

-  NMD:  6  tests  in  2+  years  before  readiness-to-rieploy  review 


Peacekeeper  program  planned  20  flight  tests  in  4+  years  (flew  19) 


The  “early  capability”  approach  demands  operational  capability  before 
system  design  is  completed  through  the  Engineering  Manufacturing 
Development  (EMD)  phase.  This  approach  is  inconsistent  with  the  complexity 
of  the  task  and  has,  thus  far,  not  accelerated  operational  capability.  Instead,  the 
added  risk  has  produced  little  discernible  benefit  and  has  actually  delayed 
operational  capability. 

The  most  convincing  evidence  of  the  risk  pressures  from  this  approach  is 
found  in  the  test  planning.  This  planning  is  characterized  by  either  less-than- 
minimal  testing  or  highly  compressed  testing  or  both. 

For  THAAD,  the  original  plan  was  2  years  to  the  first  test  flight  and  then 
almost  a  test  flight  per  month  for  the  next  2  years.  Thus  far,  the  response  to 
failures  has  been  to  reduce  the  testing  in  an  attempt  to  maintain  the  schedule. 
The  NTW  test  schedule  is  not  compressed,  but  the  number  of  planned  tests  is 
not  consistent  with  the  task.  The  Patriot  program,  which,  in  most  respects,  is 
carefully  planned  and  is  building  on  a  legacy  of  well-developed  processes,  also 
has  been  forced  into  the  less-than-minimum  test  mode. 

Current  planning  for  the  NMD  test  program  is  even  more  optimistic  than  the 
theater  HTK  programs. 

As  noted  here,  as  a  benchmark,  the  Peacekeeper  program — certainly  no 
more  technically  challenging  than  HTK— responded  to  intense  schedule 
pressure  with  a  clearly  adequate  and  well-paced  test  program  and  delivered  the 
required  capability  on  schedule. 
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Key  Judgments  (3  of  6) 
Information  Sharing 

•  Sharing  information  among  programs  and  Services  is  deemed  vital  to  the 
efficiency  and  effectiveness  in  solving  the  very  difficult  problems  of  HTK 
BMD 

•  Competition  among  programs  should  not  be  allowed  to  interfere  with  cross^tell 
of  information 

-  BMDO  needs  to  enforce  cross-tell  vigorously 

•  A  reliable  and  repeatable  process  is  needed  for  transferring  *'tribal  knowledge** 
from  one  project  management  to  another 

-  For  example,  the  design  and  engineering  processes  that  have  been  used  for 
a  long  time  in  the  Patriot  program  can  benefit  the  other  HTK  BMD 
programs 

-  Use  of  “graybeards**  for  this  purpose  is  useful  but  should  not  be  **the 
system** 


The  study  group  was  struck  by  the  similarities  among  the  challenges  and 
the  similarities  among  the  likely  solutions  to  flight  test  failures.  However,  we 
also  found  that  lessons,  approaches,  and  solutions  used  in  one  program  were 
often  not  available  for  other  programs  with  similar  challenges. 

BMD  programs  rely  heavily  on  the  “graybeard”  community  to  transfer 
“tribal  knowledge.”  While  such  groups  can  provide  useful  insights  and  advice, 
they  cannot  be  a  substitute  for  effective  formal  and  informal  cross-tell  at  the 
management  and  engineering  levels  among  programs  facing  similar 
chaDenges.  This  activity  needs  to  be  a  high-priority  responsibility  of  BMDO. 


Key  Judgments  (4  of  6) 
Developmental  Test  Philosophy 

•  The  mindset  that  risky  **key  demonstration"  tests  can  prove  readiness  for  early 
deployment  has  permeated  some  BMD  DT  programs  and  is  a  key  departure  from 
the  test  paradigm  that  has  proven  to  be  success^l  in  other  complex  programs 

•  BMD  programs  need  to  pay  more  attention  to  ground  testing,  simulation,  and 
analyses  to  reduce  known  areas  of  uix;ertainty  to  be  resolved  in  flight  tests  to  only 
those  issues  that  cannot  be  investigated  with  ground  testing.  The  more  limited  the 
flight  testing  program,  the  more  essential  it  is  to  reduce  uncertainty 

•  The  philosophy  appears  to  be  to  plan  for  a  single  test  in  each  "regime"  (e.g.,  exo, 
endo,  long-range,  short-range)  and  then  move  on 

•  There  is  a  need  to  hold  the  test  vehicle  configuration  as  constant  as  possible  for  a 
needed  series  of  tests 


The  rush  to  failure  in  flight  testing  has  been  partially  caused  by  a  fundamental 
misunderstanding  of  the  purpose  of  developmental  testing.  Some  of  these  tests 
were  treated  as  demonstrations  of  known  capabilities  where  “fly  to  verify”  was 
the  purpose.  In  practice,  the  unknowns  made  them  “fly  to  learn”  experiences. 

The  “demonstration  mindset”  was  evident  in  flight  tests  conducted  without 
complete  component  qualification  and  ground  testing.  One  program  office 
espoused  the  concept  of  “test  a  little,  learn  a  lot.”  The  drive  for  early  capability 
based  on  minimum  capability  demonstration  has  been  a  factor  in  this  “key 
demonstration”  mentality — ^ihat  is,  a  single  success  is  regarded  as  a  large  step 
forward  and  becomes  the  criteria  for  a  key  program  decision,  such  as  exercising 
an  option  to  buy  operational  missiles.  This  approach  and  mindset  are  sharp 
departures  from  experience  on  successful  flight  test  programs  that  have  followed 
the  practice  of  “learn  a  lot”  and  then  “test  to  verify.” 

BMD  programs  need  to  pay  more  attention  to  reducing  the  uncertainties  to 
only  those  issues  that  cannot  be  tested  on  the  ground  or  adequately  simulated. 

One  example  is  that  none  of  the  infrared  (IR)  HTK  programs  (THAAD,  ALI,  and 
NMD)  have  exploited  or  plan  to  exploit  existing  high-fidelity  scene  generation 
capabilities  to  exercise  their  hardware  to  the  maximum  advantage.  Test  planning 
needs  to  be  very  explicit  in  identifying  the  ground  test  and  flight  test  needs  for 
each  key  issue. 

In  general,  the  test  programs  are  designed  to  provide  a  single  shot  in  each 
operating  regime.  While  back-up  hardware  is  available —  in  most  cases — ^to 
repeat  tests,  the  single-shot  planning  produces  unrealistic  test  schedules  and 
pressures  to  move  on  despite  failures  to  achieve  test  objectives. 
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Key  Judgments  (5  of  6) 

Added  Risk  Is  Not  Working 

•  The  strategy  of  accepting  a  high  level  of  risk  to  shorten  schedule  time  has  been 
counterproductive 

-  THAAD  is  4  years  behind  schedule 

-  NTW  has  just  delayed  its  deployment  date  and  has  begun  a  risk-reduction 
program  (AU) 

-  The  path  to  NMD  operational  capability  is  largely  undefined 

•  Historically,  the  most  likely  cause  of  program  slips  has  been  high  technical 
risk 


The  study  group  was  not  surprised  to  find  that  accepting  higher  risk  is  not 
accelerating  fielded  capability.  The  virtually  universal  experience  of  the  study 
group  members  has  been  that  high  technical  risk  is  not  likely  to  accelerate 
fielded  capability.  It  is  far  more  likely  to  cause  program  slips,  increased  costs, 
and  even  program  failure. 
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Key  Judgments  (6  of  6) 

The  NMD  Program 

•  Schedule  and  cost  pressures  on  NMD  have  created  a  planning  environment  at 
least  as  optimistic  as  that  which  has  lead  to  test  failures  and  delays  in  TMD 
programs 

•  The  NMD  program  consists  of  a  series  of  very  difficult  challenges 

-  Although  NMD  activity  has  been  ongoing  for  a  long  time,  there  has  not 
been  a  coherent,  consistent  path  and  a  realistic  plan  leading  to  a  deployed 
system 

-  There  are  high  schedule  rislcs  and  inadequate  test  assets  and  testing 
planned  in  the  3  4  3  fonnulation 

-  In  the  judgment  of  the  study  group,  successful  execution  of  the  3  4  3 
fonnulation  on  the  planned  schedule  is  highly  unlikely.  The  program  will 
benefit  from  the  earliest  possible  restructuring  to  contain  the  risk 


For  NMD,  the  schedule  and  cost  pressures  inherent  in  the  3  +  3  formulation 
and  the  system  requirements  are  inherently  even  more  severe  than  those  for  the 
TMD  programs  that  have  experienced  excessive  flight  test  failures. 

To  succeed,  the  NMD  program  must  meet  a  series  of  formidable 
challenges.  The  effort  to  meet  these  challenges  must  emanate  from  a  clear  set 
of  requirements,  consistent  resource  support  (which  includes  an  adequate 
number  of  test  assets),  well-defined  milestones,  and  a  rigorous  test  plan.  The 
study  group  believes  that  current  NMD  program  is  not  characterized  by  these 
features  and  is  on  a  high-risk  vector.  It  will  benefit  from  the  earliest  possible 
restructuring  to  a  more  achievable  set  of  goals. 
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Recommendations  for  Reducing  Flight  Test  Risk 
and  Fielding  Capability  Expeditiously 


The  study  group’s  initial  attempt  to  distinguish  between  best  practices  and 
specific  program  recommendations  became  counterproductive.  Hence,  the 
following  slides  provide  specific  recommendations  for  implementing  best 
practices  in  BMD  programs.  We  have  emphasized  those  practices  most 
relevant  to  the  NMD  program. 
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Recommendations  for  Reducing  Fiight  Test  Risk 
AH  BMD  Programs  (1  of  4) 

•  Put  the  programs  on  realistic  schedules  now.  Do  not  wait  for  further  failures 

-  Schedules  must  be  consistent  with  historical  programs 

~  Schedules  can  be  more  aggressive*  but  ofdy  if  justified  by  the  new  processes 
or  approaches  that  support  shorter  development  times 

-  Accelerating  schedules  by  simply  adding  risk  carries  a  very  high  risk  of 
failure 

•  Focus  intensely  on  sequentially  demonstrating  that 

-  The  technology  for  the  selected  approach  to  HTK  is  ready  for  operational 
systems 

-  The  technology  is  operationally  reliable 

-  The  technology  is  robust  to  at  least  the  first  layer  of  countermeasures 

•  Require  a  detailed  test  plan  that  includes  a  full  set  of  ground  simulation  and  tests 
(to  include  HWIL)  tied  to  each  flight  test  objective.  Include  the  test  facility 
requirements  and  plan 


These  first  sets  of  recommendations  apply  to  all  BMD  programs,  including 
NMD. 

To  reduce  the  pressure  on  the  programs  “to  shoot”  before  progressing  to  a 
reasonable  probability  of  success,  BMDO  must  put  its  programs  on  realistic 
schedules  before  failures  occur.  These  schedules  should  be  consistent  with 
those  of  past  successful  programs.  Aggressive  schedules  should  be  allowed 
only  if  they  can  be  justified  by  new  processes  or  approaches  that  will  support 
accelerated  schedules. 

For  the  near  term,  the  BMDO  programs  should  focus  on  demonstrating  that 
HTK  technology  is  viable  and  that  HTK  against  simple  targets  can  be  achieved 
reliably.  Only  after  this  has  been  demonstrated  can  the  programs  continue  to 
demonstrate  that  the  proposed  weapon  system  is  operationally  feasible.  As 
noted  in  the  Findings  Section,  some  requirements  on  systems  result  from  a 
desire  to  demonstrate  operational  capability  before  the  system  design  and 
development  have  been  completed.  Relaxing  these  requirements  would  raise 
the  probability  of  success  in  the  first  step  of  demonstrating  HTK.  Finally,  the 
weapon  system  must  demonstrate  that  it  is  robust  to  a  first  layer  of 
countermeasures. 

While  all  the  programs  have  a  test  plan,  most  programs  do  not  make  use  of 
the  ground  simulation  and  testing  warranted  by  the  difficulty  of  the  in-flight 
task. 
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Recommendations  for  Reducing  Flight  Test  Risk 
All  BMD  Programs  (2  of  4) 

•  Ensure  that  test  planning — ground  and  flight — ^reduces  the  test  flight 
uncertainties  to  only  those  issues  that  must  be  resolved  in  flight.  Flight  test 
treatment  of  other  technical  issues  should  be  verification,  not  surprises 

•  Eliminate  the  demand  for  fielded  operational  capabilities  in  advance  of  EMD. 
Regardless  of  the  desire  for  '*early*'  capability,  this  approach  is  unlikely  to  be 
productive  for  programs  of  this  complexity 

•  Ensure  that  the  Services  are  responding  to  Department  of  Defense  (DoD) 
priority  decisions  rather  than  inconsistent  Service-assigned  '‘urgency’' 


As  noted  in  the  Key  Judgments,  the  flight  test  risk  should  be  reduced  to 
only  those  uncertainties  that  must  be  resolved  in  flight.  Other  issues  should  be 
resolved  in  ground  testing.  Right  testing  should  then  be  used  for  verification. 

The  attempt  to  achieve  an  early  operational  capability  before  EMD  is 
workable  for  systems  and  capabilities  that  are  reasonably  well  in  hand.  For 
complex,  demanding  tasks  (i.e.,  HTK)  that  have  yet  to  be  demonstrated 
adequately,  the  drive  for  early  capability  is  proving  to  be  counterproductive. 

The  NTW  program  seems  to  have  modified  significantly  the  demand  for 
early  capability;  however,  the  THAAD  program  continues  to  pursue  this 
capability  with  undiminished  zeal.  The  THAAD  program  should  be  relieved 
of  this  requirement,  and  the  energy  and  resources  should  be  channeled  to  the 
EMD  program. 

Program  office  briefings  for  all  the  HTK  EMD  programs  stressed 
operational  “urgency”  as  justification  for  accelerated,  high-risk  approaches. 
For  some  of  these  programs,  the  operational  “urgency”  generated  by  the 
Service  seems  out  of  proportion  with  the  joint  priorities  and  the  program 
resources. 


Recommendations  for  Reducing  Flight  Test  Risk 
All  BMD  Programs  (3  of  4) 

•  Need  more  rigorous  ground  testing  that  uses  the  best  available  simulations  and 
test  facilities 

-  BMDO  should  conduct  a  comprehensive  review  of  current  and  planned 
capabilities  to  ensure  that  facilities  are  adequate  for  testing  and  that 
funding  will  be  available  to  support  any  necessary  upgrades 

-  BMDO  should  insist  upon  more — and  more  rigorous — HWIL  testing  of 
IR  HTK  systems 

-  Need  an  approach  similar  to  captive-carry,  which  permits  sequential, 
repetitive,  non-destructive  test  (NDT)  and  find  and  fix  testing  of  critical 
flight  software  and  hardware,  such  as  that  in  the  HTK  vehicle 


To  ensure  adequate  HWIL  test  facilities  and  the  best  use  of  available 
facilities,  BMDO  should  conduct  a  comprehensive  review  of  current  ground 
test  capabilities,  including  capabilities  that  complement  HWIL  ( e.g.,  tethered 
systems  and  hover  test  facilities). 

Programs  should  embrace  a  testing  approach  that  provides  for  sequential, 
replicable,  non-destructive  ground  tests,  with  simulations  and  ground  test 
facilities  providing  the  supporting  capabilities.  End-to-end  system  simulation 
is  vital  to  reduce  flight  test  risk,  and  it  should  include  more  use  of  HWIL 
testing  for  critical  flight  hardware. 

KV  ground  testing  should  include  realistic  scene  generation  as  part  of  a 
HWIL  capability  for  testing  end-to-end  KV  performance. 

If  BMDO  finds  that  facilities  are  not  adequate  for  providing  such 
capabilities  to  the  extent  needed  for  supporting  the  BMD  programs,  BMDO 
should  place  a  high  priority  on  a  coherent,  near-term  investment  program  to  fill 
the  gaps. 

A  continuing  program  should  also  be  in  place  to  upgrade  capabilities  as 
needed. 
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Recommendations  for  Reducing  Flight  Test  Risk 
All  BMD  Programs  (4  of  4) 

•  BMDO,  strongly  supported  by  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  (USD(A&T)),  needs  to  play  a  more  compelling  role  in  BMD  pro¬ 
grams  execution 

•  BMDO  should  establish  the  following  driving  philosophy  for  flight  testing  for 
all  HTK  programs: 

-  A  complete  environmental  specification  must  be  prepared  so  that  all 
critical  components  are  fully  qualified  for  expected  flight  conditions,  with 
adequate  margins  to  handle  the  unexpected 

-  Program  office  certincation,  prior  to  Flight  Readiness  Review  (FRR),  that: 

•  All  items  have  been  qualified  to  at  least  3  dB  above  predicted 
environments 

•  All  items  have  undergone  rigorous  ground  testing 

•  IR  HTK  seekers  have  been  tested  in  rigorous  high-fidelity  HWIL 
simulations 


BMDO  must  take  a  more  active  role  in  ensuring  adequate  preparation  for 
flight  testing.  This  will  require  aggressive  BMDO  initiatives  and  strong 
USD(A&T)  support. 

For  example,  BMDO  should  establish  the  driving  philosophy  indicated 
here.  The  environmental  specification  should  ensure  that  critical  components 
will  have  an  adequate  margin  to  deal  with  unexpected  conditions. 

A  formal  process  is  needed  to  ensure  full  certification  of  the  system  before 
each  flight  test.  This  process  should  include  rigorous  ground  testing  and 
software-in-the-loop  (SWIL)  and  HWIL  simulations. 
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Recommendations  for  Reducing  Flight  Test  Risk 
NMD  (1  of  2) 

•  Acknowledge  that  the  3  -f  3  program  schedule  is  very  high  risk 

-  The  start  time  for  3  +  3  was  October  1996;  however,  the  integrating 
contractor  responsible  for  detailed  program  planning  is  not  being  selected 
until  spring  1^8 

-  The  program  should  be  restructured  now  to  eliminate  unrealistic  expectations 

•  Whether  or  not  BMDO  continues  to  pursue  3  -f  3: 

-  Restructure  test  flight  programs  to  allow  time  between  tests  to  ensure  that 
testing  is  completely  sequential — that  is,  allow  adequate  time  between  test 
flights  to  correct  deficiencies  and  ensure  that  adequate  assets  are  available  for 
repeat  tests 

-  Increase  funding  for  test  assets  and  increase  the  planned  testing  to  include  a 
planned  back-up  test  for  the  sensor  fly-by  (la/2)  and  the  intercept  attempts 
(3/4) 

-  Increase  funding  support  for  ground  tests  to  ensure  capability  for 
comprehensive  subsystem  and  end-to-end  system  testing 

-  Continue  to  fund  key  technology  development  for  the  earliest  system  and  for 
follow-on  system  capability  improvements  moving  from  demonstrated 
operational  utility  to  increasingly  robust  counter-countermeasures 


Expecting  the  required  development  and  testing  for  deployment  readiness 
to  be  completed  by  the  end  of  2000  is  unrealistic.  The  NMD  program  should 
be  restructured  now  to  provide  for  adequate,  sequential  development  and 
testing. 

While  the  3  +  3  program  is  not  a  UOES  program  in  the  sense  used  in 
THAAD,  it  carries  similar  potential  for  interference  with  an  orderly  operational 
system  program.  In  particular,  the  3-year  development  program  is  driven  by 
the  need  to  be  ready  to  deploy  3  years  later.  This  has  led  to  a  highly 
compressed  and  less-than-minimum  flight  and  ground  test  program. 

In  any  case,  these  specific  recommendations  will  reduce  the  NMD  risk. 

The  philosophy  of  the  3  +  3  program  calls  for  continued  evolution  of 
capabilities  in  the  period  between  a  successful  readiness  review  and  the  actual 
decision  to  deploy  a  system.  This  approach  requires  a  continuing  and  vigorous 
key  technology  development  program  to  ensure  that  the  program  continues  to 
evolve  to  meet  the  changing  capability  needs  over  time. 
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Recommendations  for  Reducing  Fiight  Test  Risk 
NMD  (2  of  2) 

•  To  reduce  the  risk  associated  with  the  challenging  Kinetic  Kill  Vehicle  (KKV) 
perfonnance  issues,  DMDO  should  mandate  that  the  NMD  program^ 
operating  through  the  lead  system  integrator— have  both  of  the  candidate 
KKVs  rigorously  tested  at  the  Kinetic  Kill  Vehicle  Hardware*in-the>Loop 
Simulation  (KHILS)  facility  to  determine  their  suitability  for  NMD 


In  addition  to  the  emphasis  in  earlier  Key  Judgments  and  recommendations, 
KKV  performance  warrants  extraordinary  attention.  As  a  minimum,  BMDO 
should  demand  extensive  testing  of  candidate  KKV  systems  in  the  most  capable 
facility — ^the  KHILS. 
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Findings: 

Responses  to  Speciflc  Questions 
in  the  Tasking 


This  section  responds  directly  to  speciflc  questions.  Some  flndings  are 
repetitive  points  that  have  already  made  in  the  Key  Judgments  and  the 
recommendations;  however,  the  repetitive  points  tend  to  be  those  most  in  need 
of  further  emphasis. 
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Findings  (1  of  8) 

Review  of  T&E  Paradigms  Used  in  Past  Successful  Programs 

•  Pirogram  experience  verifies  that  success  depends  on  following  proven  paradigms: 

-  Realistic  system  requirements 

~  Schedule  plans  based  either  on  successful  experience  or  supported  by  the  use 
of  newer  processes  with  shorter  time  spans 

-  Design  and  systems  engineering  discipline  with  adequate  design  margins 

-  Full  component  qualification 

-  Adequate  simulation  and  other  ground  testing 

-  Adequate  numbers  of  flight  tests  and  sequential  testing 

-  £nd-to>end  flight  readiness  evaluations 

-  Top-quality  people  in  government  program  offices  and  on  contractor  teams 

•  In  addition,  the  nature  of  BMD  programs  demands  additional  emphasis  on: 

-  High-fidelity  simulations 

-  HWIL  testing;  need  new  approaches  that  allow  for  repetitive  and  iterative 
find  and  fix  testing  of  expendable  flight  hardware— the  BMD  equivalent  of 
captive-carry  for  air-to-air  missiles 

-  Demonstrating  and  maturing  HTK  technologies 


The  collective  experience  of  the  study  group  members  suggests  that  a  successful 
flight  test  program  starts  with  a  realistic  schedule.  There  must  be  the  time  and  the 
commitment  for  a  system  to  pass  through  a  disciplined  design  process  and  the 
painstaking  intervening  steps  that  make  the  system  ready  for  each  sequential  flight 
test.  Our  collective  experience  also  suggests  that  test  dates  have  to  be  driven  more 
by  successful  completion  of  events  than  by  the  calendar. 

End-to-end  flight  readiness  evaluations  are  essential.  For  BMD  programs,  these 
evaluations  have  too  often  been  most  evident  and  most  thorough  after  a  failure 
instead  of  before  the  flight  test  These  reviews  consistently  verified  the  validity  of 
the  historic  flight  test  paradigm. 

In  addition  to  these  historic  paradigms  that  produced  successful  flight  tests  of 
complex  systems,  the  characteristics  of  HTK  programs  demand  increased  emphasis 
on  certain  aspects  of  the  paradigm  (e.g.,  design  margins,  full  qualification  of 
components,  careful  analysis  of  critical  functions  and  components,  thorough  ground 
end-to-end  tests,  and  so  forth).  The  challenge  of  HTK  also  warrants  additional 
emphasis  on  HWIL  testing  and  high-fidelity  simulations. 

The  historic  T&E  paradigm  is  still  valid  and  is  essential  to  successful  flight  test 
programs,  but  it  is  not  being  adequately  followed  in  BMD  programs.  This  paradigm 
is  affordable  for  HTK  systems  and  is  far  less  costly  than  the  current,  riskier 
approaches  that  produce  flight  test  failures,  program  delays,  and  possible  program 
failure. 
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Findings  (2  of  8) 

Review  of  Peacekeeper  Missile  Program  and  Spacecraft  Programs 

•  Followed  the  principles  listed  in  the  previous  slide 

•  Some  key  Peacekeeper  missile  program  characteristics 

-  Constant,  very  higb*priority  national  program 

-  A  new  generation  of  ICBM  that  is  building  on  a  history  of  past  successes 

-  Rigorous  perfonnance  criteria 

-  6  years  planned  from  selection  of  contractor  to  first  flight  (actually  took 
6  1/2  years) 

-  Despite  time  pressures,  high  priority  on  adequate  quantity  of  tests  and 
adequate  time  between  tests.  Planned  20  test  flights  in  44  years 

-  Completed  on  schedule  and  achieved  performance.  18  test  flights 
required 

•  Successful  spacecraft  programs  include: 

-  Redundancy  and  design  margins  for  critical  functions 

-  Rigorous  qualiflcation  of  components  and  subcomponents 

-  End>to>end  preflight  checkout  capabilities 


The  Peacekeeper  program  provides  a  prototypical  example  of  a  successful, 
high-priority  development  program.  It  is  particularly  noteworthy  that  there 
were  intense  pressures  for  early  initial  operational  capability  (IOC)  for 
Peacekeeper— from  the  user,  from  the  DoD  leadership,  and  from  the  National 
leadership. 

Further,  the  Peacekeeper  program  was  a  new  generation  of  a  proven 
weapon  (the  ICBM),  which,  while  pushing  the  state  of  the  art,  did  not  demand 
any  fundamental  technology  not  already  demonstrated. 

Even  so,  the  program  took  6  1/2  years  from  start  to  first  flight  and  had  a 
rigorous,  disciplined,  flight  test  program  with  adequate  time  between  tests  to 
analyze  results  before  the  next  test — a  thoroughly  sequential  approach.  The 
program  reached  IOC  on  time  and  with  less  than  the  planned  number  of  test 
flights. 

Highly  relevant  lessons  also  apply  to  BMD  programs  from  a  myriad  of 
spacecraft  programs:  the  need  for  very  high  reliability  in  critical  systems, 
rigorous  component  and  system  qualification,  and  end-to-end  ground  checkout 
before  the  system  is  launched. 
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Findings  (3  of  8) 

Sprint  Missile  (SAFEGUARD) 

•  High-priority  program 

•  Intense  time  pressures 

•  Program  manager  reported  directly  to  the  Director  of  Defense 
Research  and  Engineering  (DDR&E)  (Dr.  Foster) 

•  Test  planning  and  record 

-  42  missile  tests  in  5  years 

•  23  successes  (S) 

•  9  partial  successes  (PS) 

•  10  unsuccessful  (U) 

-  First  10  shots:  S.  PS.  U.  U.  U,  U.  PS.  PS.  PS.  S 


The  Sprint  Missile  program,  as  an  element  of  the  SAFEGUARD  system,  is 
another  example  of  a  high-priority  program  that  was  executed  under  intense 
schedule  pressures. 

With  this  highly  compressed  test  schedule,  the  first  10  tests  were 
characterized  by  a  high  failure  rate.  However,  this  failure  rate  was  made 
tolerable  by  the  extensive  planned  series  of  tests  that  followed. 
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Findings  (4  of  8) 

Some  Characteristics  of  Present  HTK  BMD  Programs 

•  Failures  were  often  described  to  the  study  group  as  random  or  anomalous 

•  Failures  are  rarely  random  or  anomalous 

-  THAAD  failures  resulted  from  poor  design,  fabrication,  management,  and 
quality  control 

-  Navy  CTV-1  failure  resulted  from  poor  design 

•  Root  causes  of  the  failures  include; 

-  Unrealistic  schedules 

-  Underestimation  of  the  difficulty  of  achieving  HTK 

-  Inadequate  component  and  system  qualification  and  ground  test 


The  study  group  heard  repeated  references  to  “random”  failures.  However, 
few,  if  any,  of  these  failures  were  “random” — a  statistical  matter.  They  were 
caused  by  poor  design,  test  planning,  and  preflight  testing  deficiencies;  poor 
fabrication;  poor  management;  and  lack  of  rigorous  government  oversight. 

The  tendency  of  the  government  and  program  managers  to  trivialize  the 
causes  of  these  costly  failures,  combined  with  the  aggressive  schedule 
discussed  on  the  previous  slide,  has  led  to  a  “rush  to  failure.” 

We  felt  that  the  program  managers — ^both  government  and  contractor — 
underestimated  the  degree  of  difficulty  in  achieving  HTK.  The  fact  that  the 
contractor  simulations  often  predicted  HTK  100  percent  of  the  time  gave  us  the 
impression  that  the  contractors  routinely  underestimate  the  many  things  that 
can  go  wrong.  We  felt  that  this  lack  of  appreciation  for  the  complexity  of  the 
task  continued  after  experience  should  have  provided  compelling  evidence  to 
the  contrary. 


Findings  (5  of  8) 

Examine  How  Best  To  Identify  Likely  Failure  Modes; 
Recommend  Practices  and  Preflight  Test  Options 

•  Success  depends  on  a  detailed,  well-documented,  integrated  test  plan  vs.  a  plan 
for  a  series  of  demonstrations 

•  HTK  program  planning  has  been  characterized  by  inadequate  and/or 
compressed  flight  and  ground  testing.  Requires  an  up-front  commitment  to 
repeat  tests  when  needed 

•  Need  adequate  investment  in  spare  systems  and  targets  to  repeat  tests  as  needed 

-  Need  to  avoid  the  temptation  to  declare  success  and  move  on  to  the  next 
test  when  key  test  objectives  have  not  been  achieved 

•  Compressed  test  schedules  lead  to  parallel  testing  that  assumes  a  high  degree  of 
success  and  does  not  accommodate  the  need  to  incrementally  find,  Hx,  and 
retest  when  problems  occur 

•  An  acceptable  probability  of  success  depends  on  a  very  rigorous  and 
disciplined  subsystem  and  system  development  program  with  well-established 
and  rigorous  Program  Design  Reviews  (PDRs),  Critical  Design  Reviews 
(CDRs),  qualification  tests,  and  so  forth.  The  importance  of  this  approach 
grows  exponentially  with  reduced  numbers  of  flight  tests 


Success  in  flight  testing  has  depended  on  detailed,  well-documented, 
integrated  test  planning  vs.  depending  on  a  series  of  leap-ahead 
demonstrations. 

In  contrast  to  this  need,  BMD  HTK  program  planning  has  been — and  much 
of  it  still  is — characterized  by  inadequate  and  compressed  flight  and  ground 
testing.  Much  of  the  testing  is  in  parallel,  with  inadequate  time  between  tests 
to  correct  problems  before  the  next  test.  This  situation  is  exacerbated  by 
inadequate  provisions  to  repeat  tests  that  fail. 

To  the  extent  that  cost  or  other  factors  demand  minimum  test  flights,  the 
importance  of  the  other  aspects  (design  margins,  component  qualification, 
ground  tests,  preflight  review,  and  so  forth)  becomes  even  more  critical. 

Regardless  of  the  approach,  when  developing  new  and  unprecedented 
capabilities,  flight  test  failures  will  occur.  A  prudent  program  will  anticipate 
and  account  for  this  reality.  Failures  can  be  minimized  with  incremental, 
sequential  testing;  however,  they  cannot  be  eliminated. 
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Findings  (6  of  8) 

Examine  How  Best  to  Identify  Likely  Failure  Modes; 
Recommend  Practices  and  Preflight  Test  Options 

•  Need  more  rigorous  ground  testing  that  uses  the  best  available  simulations  and 
test  facilities 

-  HWIL  for  critical  functions 

-  Need  an  approach  similar  to  captive-carry,  which  permits  sequential, 
repetitive.  NDT  and  fuid  and  fix  testing  of  critical  flight  software  and 
hardware,  such  as  that  in  the  HTK  vehicle 

-  Realistic  scene  generation 

-  Need  end-to-end  system  simulation  to  include  maximum  critical  HWIL 

•  Ground  testing  emphasis  expanded  greatly  following  failures.  Needs  to  be  an 
up-front  requirement,  not  a  response  to  failure 


The  study  group  found  significantly  increased  emphasis  on  ground  testing 
following  test  failures.  We  also  found  an  array  of  existing  capabilities  for 
simulation  and  ground  testing,  including  concepts  that  approach  end-to-end 
preflight  testing.  We  did  not  find  consistently  coherent  plans  to  make  the  best 
use  of  this  array  of  capabilities  nor  a  rigorous  analysis  of  remaining 
deficiencies  and  programs  to  HU  those  gaps. 

We  found  approaches  that  might  fill  the  function  that  captive-carry  fills  for 
air-to-air  and  air-to-surface  missiles:  providing  find-and-fix  opportunities  for 
system  interfaces  and  critical  functions.  These  approaches  included  tethered 
systems  with  dynamic  scene  generation  and  expanded  hover  testing.  This 
capability  is  critical  for  reversing  the  record  of  failures  in  HTK  BMD 
programs.  Again,  we  did  not  find  the  use  or  planned  use  of  these  approaches 
in  BMD  programs. 
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Findings  (7  of  8) 

Examine  How  Best  to  Identify  Likely  Failure  Modes  ; 

Recommend  Practices  and  Preflight  Test  Options 

•  THAAD  flight  test  experience  indicates  that  the  initial  design  and  fabrication 
were  not  subjected  to  adequately  disciplined  engineering  practices  and 
rigorous  quality  control 

•  The  NMD  program,  which  will  depend  on  an  integration  contractor  for  these 
functions,  needs  to  pay  intense  attention  to  these  lessons  from  the  THAAD 
program 

•  Time  pressures  continue  to  inhibit  the  right  level  of  discipline  in  the  THAAD 
program 

•  The  NMD  program  planning  creates  a  high  risk  that  these  same  pressures  can 
drive  NMD  design  and  test  philosophy 


Since  the  THAAD  program  has  produced  the  most  flight  test  experience  to 
date,  it  also  provides  the  richest  source  of  lessons  learned  for  the  NMD 
program. 

Numerous  reviews  and  the  Integrated  Product  Team  (IPT)  process  have 
reported  that  initial  design  and  fabrication  were  not  subjected  to  adequate 
discipline  and  quality  control.  Further,  the  THAAD  program  office  has  also 
expressed  these  concerns  at  various  times. 

To  a  large  extent,  the  NMD  program  will  depend  on  an  integration 
contractor  for  these  functions.  The  NMD  program  will  need  a  rigorous 
approach  for  ensuring  that  the  early  THAAD  experience  is  not  repeated  in 
NMD.  The  3  +  3  approach  leaves  no  room  for  the  failures  of  proven 
technology  that  can  be  very  nearly  eliminated  with  discipline  and  quality 
control. 

As  already  suggested  several  times,  schedule  pressure  can  be  a  powerful 
opponent  of  rigorous  design,  fabrication,  and  test  discipline.  The  3  +  3 
approach  is  likely  to  produce  intense  schedule  pressures. 
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Findings  (8  of  8) 

HWIL  Test  Facilities 

•  The  BMDO>develop6d  HWIL  facility  (KHILS)  is  underused 

•  Each  Service  is  also  investing  in  its  own  HWIL  test  facility,  with  little  regard 
for  available  capabilities  in  other  Services  or  agencies 

•  The  technology  needed  to  develop  such  facilities  (e.g.,  scene  generation  and 
projection)  is  becoming  less  costly  and  more  capable  every  year 

•  THAAD  and  NMD  do  not  appear  to  be  leveraging  the  capabilities  in  these 
facilities 

-  THAAD  has  not  subjected  the  seeker  for  FT-8  to  the  most  rigorous  testing 

-  NMD  does  not  appear  to  be  planning  such  HWIL  testing  before  the 
interceptor  down-select 


Even  though  an  extensive  array  of  simulation  and  test  facilities  are 
available,  the  study  group  was  unable  to  find  a  comprehensive,  coherent  plan 
for  their  use  and  further  development — ^particularly  for  HWIL  testing  of 
critical  components. 

A  comprehensive  review  of  the  largely  Service-and  contractor-developed 
capabilities  is  urgently  needed  to  provide  a  coherent  plan  for  current  use  and 
for  a  DoD  investment  plan  to  fill  fiiture  needs. 

We  found  that  the  THAAD  and  NMD  programs  were  not  making 
maximum  use  of  existing  facilities.  The  THAAD  contractor  felt  that  the  seeker 
for  FT-8  did  not  need  to  be  subjected  to  the  most  rigorous  level  of  testing, 
because  of  the  large  target  size  in  FT-8.  The  NMD  program  was  also  not 
planning  use  of  the  KHILS  Facility  to  learn  more  about  the  two  candidate 
seekers  in  its  program. 
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Specific  Program  Reviews 


This  section  contains  selected  information  from  specific  program  reviews. 

The  study  group  received  extensive  reports  on  these  programs  from 
government  and  contractor  program  managers.  This  section  focuses  on  the 
experience  and  lessons  learned  that  arc  the  basis  for  the  key  judgments  and 
findings  in  the  previous  sections. 
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Peacekeeper  ICBM  Program 

High  Pressure,  Careful  Planning,  Deliberate  Execution 


•  National  program,  national  priority 

•  Strong  user  demand  for  an  early  IOC.  Still  6  years  from  start  to  first  flight 

•  Strong  requirements  focus — both  operational  and  test  requirements 
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•  Phased  approach  to  reduce  flight  test  risk 

•  Full  qualiflcation  of  individual  components 

•  Ground  tests  to  demonstrate  subsystem  performance 

•  Three  phases 

-  Phase  1:  Does  Peacekeeper  work?  Address  technical  risk  areas  early 

-  Phase  2;  How  well  does  Peacekeeper  work  as  a  weapon  system?  Define 
final  configuration  and  operational  procedures 

-  Phase  3:  Is  Peacekeeper  ready  for  IOC? 


The  Peacekeeper  ICBM  program  exemplifies  a  complex,  high-  priority 
program  in  which  a  highly  successful  flight  test  program  was  planned  and 
executed  in  an  environment  of  intense  pressures  for  an  early  ICX^. 

The  starting  point  was  a  strong  and  uncompromising  requirements  focus 
both  for  operational  capabilities  and  test  planning. 

The  test  program  was  comprehensive,  incremental,  and  sequential  and 
provided  time  between  tests  to  analyze,  fix,  and  prepare  for  the  next  test. 
Partial  failures  did  occur;  however,  in  each  case,  the  cause  was  confidently 
determined  to  have  been  aided  by  uncompromised  instrumentation. 

The  program  to  control  flight  test  risk  followed  the  successful  historic  test 
paradigm:  full  qualification  of  components  and  subsystems,  comprehensive 
ground  tests,  and  disciplined  sequential  testing  with  time  to  fix  deficiencies 
before  the  next  test. 
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Peacekeeper  ICBM  Program 
System  Test  Program 

•  Based  on  a  full  Test  and  Evaluation  Master  Plan  (TEMP)  and  an  Integrated 
Test  Plan  (TIP) 

•  Tbe  Peacekeeper  FTP  was  the  basis  for  all  test  planning  analysis  (TPA) 

-  Defined  319  major  ground  tests  and  20  flight  tests  for  the  development, 
test»  and  evaluation  (DT&E)  and  tbe  initial  operational  test  and  evaluation 
(lOT&E)  phase 

-  All  test  facilities,  instrumentation,  and  test-unique  accommodations  on  tbe 
operational  system  were  based  on  the  FTP  and  the  detailed  TPA 

-  Instrumentation  and  test-unique  accommodations  were  among  the  most 
challenging  needs 

•  Tbe  TEMP  also  required  tbe  program  office  to  establish  and  chair  test 
planning  working  groups  in  which  project  officers,  project  engineers, 
contractors,  tbe  customer,  and  the  independent  test  organization  participated 
fully — including  the  resolution  of  anomalies 


The  system  test  program  received  intense  attention,  starting  with  a  detailed 
TEMP  and  ITP  that  included  detailed  ground  and  flight  test  requirements. 

All  test  activities,  planning,  execution,  and  review  emanated  from  the 
TEMP  and  the  ITP. 

Test  planning  working  groups  were  key  players  in  flight  test  decisions. 
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THAAD 

•  Purpose:  defend  against  medium-  to  long-range  TBMs  intercepting  low-cndo 
through  exo-atmospheric  targets  with  an  HTK  warhead 

•  Systems 

-  New  ground-based  radar  (GBR) 

-  Launchers  on  a  Palletized  Load  System  (PLS)  truck 

-  Command  and  control  (C2)  system 

•  Tactical  operations  station 

•  Launch  control  station 

-  New  missile  with  an  HTK  KKV 


The  THAAD  system  consists  of  a  new  radar,  launcher  system,  a  complex 
C2  system,  and  a  new  missile  that  incorporates  complex,  unproven  technology. 
This  missile  is  planned  to  operate  in  altitude  regimes  from  the  low  endo- 
atmospheric  to  the  exo-atmospheric. 
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THAAD  Flight  Test  Objectives  and  Results 
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The  THAAD  flight  test  program  encountered  difficulties  beginning  with 
the  second  test. 

The  causes  of  failure  in  these  flight  tests  were  found  to  be  in  subsystems 
usually  considered  to  be  low  risk.  The  failures  typically  were  caused  by  poor 
design  and  fabrication,  inadequate  ground  checkout  discipline,  and  pressures  to 
move  on  to  the  next  step.  For  FT-4,  the  cause  of  the  failure  could  have  been 
discovered  with  checkout  processes  that  are  fairly  standard  for  systems  of  this 
type. 

The  failures  led  to  extensive  reviews.  These  reviews  identified  important 
shortcomings  in  design  and  fabrication  discipline,  test  planning,  ground 
testing,  and  preflight  review. 
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THAAD:  The  Program  Environment 

Intense  Time  Pressure — Rush  to  Failure 

•  Intense  time  pressure  on  a  UOES  capability.  Mismatch  between  department*  s 
priorities  and  reality  of  the  program  challenge 

•  Higher  risk  has  been  justiiied  by  urgent  fielding  requirement.  Test  preparation 
and  instnimentation  decisions  influenced  by  desire  for  early  decision  on  the 
40  UOES  missiles 

•  Myriad  leadership  and  management  concerns  identified  by  the  program  office 
and  independent  reviews  indicated  that  the  time  pressures  bad  become 
counterproductive.  The  program  needed  fundamental  reprioritizing 

•  Multiple  additional  external  factors  contributing  to  risk 

~  Funding 

-  Political  pressures 

-  Test  range  adequacy  and  availability 

-  Targets 


While  the  justification  for  the  time  pressure  was  the  users’  need  for 
capability,  the  UOES  approach  to  satisfying  that  need  seems  to  have  stemmed 
from  seriously  underestimating  the  difficulty  of  the  task. 

A  conscious  decision  was  made  to  trade  off  technical  risk  against  the 
urgent  need.  This  decision  resulted  in  a  program  plan  that  would  buy 
operational  missiles  and  field  an  operational  system  of  a  type  that  had  never 
been  fielded  before  accomplishing  EMD.  This  decision  also  led  to  serious 
compromise  of  the  test  missiles,  requiting  that  the  test  missiles  be  essentially 
operational  rounds  with  whatever  instrumentation  could  be  accommodated. 

Following  the  early  failures  and  continuing  through  the  latest  failure, 
numerous  in-depth  reviews  (both  internal  and  external)  have  identified  the 
causes  of  the  individual  failures  and  the  deeper  underlying  causes.  However, 
there  was  little  or  no  relief  from  the  time  pressure.  The  early  program 
presentations  to  the  study  group — even  after  the  failure  of  FT-7 — emphasized 
the  urgency  of  conducting  FT-8,  with  minimum  delay  and  virtually  no 
increased  ground  tests.  Subsequent  decisions  show  some  change  to  a  more 
deliberate  approach  but  with  a  continuing  commitment  to  the  UOES  approach. 
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THAADUOES  System 

A  Symptom  of  Heavy  Schedule  Optimism 

•  Strategic  Defense  Initiative  Orgaiuzation  (SDIO)-proposed  and  Congress ionally 
mandated  1996  capability  date 

•  Essentially  demanded  an  operational  capability  before  the  missile  was  designed 

•  THAAD  battalion  at  Fort  Bliss  was  stood  up  in  June  1995 

-  2  radars  and  battle  management/command,  control,  communications, 
computers,  and  intelligence  (BM/C4I)  suites 

-  4  launchers  and  40  missiles  to  be  purchased  after  first  intercept 

•  THAAD  demonstrationAfalidation  (Dcm/Val)  to  provide  UOES 

-  48-month  contract 

-  First  flight  in  24  months 

-  20  flights  (-  1/month) 

•  Required  that  test  missiles  be  certified  rounds  rather  than  highly  instrumented 
research  and  development  (R&D)  missiles 

•  Demanded  highly  parallel  testing  vs.  a  sequential  fmd  and  fix  approach 


The  UOES  concept  was  originated  within  SDIO  and  was  included  as  part 
of  the  THAAD  program  at  its  inception.  It  was  later  written  into  law  by  the 
Congress.  It  requited  that  an  early  operational  capability — to  include  a  ready 
unit — ^be  stood  up  in  48  months.  This  necessarily  led  to  an  emphasis  on  those 
issues  that  have  to  do  with  operational  capabilities;  operational  rounds,  an  all- 
up  C2  system,  trained  soldiers,  and  so  forth.  This  compromised 
accommodating  the  best  practices  for  test  missiles  and  the  test  program. 

The  study  group  found  that  program  managers,  based  upon  their  previous 
experiences,  assumed  that  the  long  poles  were  the  radar,  C2  system,  and 
trained  soldiers.  They  did  not  anticipate  that  guiding  the  missile  to  hit  a  target 
would  prove  to  be  the  most  daunting  task.  Again,  this  is  evidence  of 
underestimating  the  difficulty  of  performing  HTK  intercepts. 
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THAAD  Program  Schedules 

Facing  Reality  but  With  Continuing  Optimism 
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Mid-FY  96  Change  (after  the  4tb  consecutive  failure  to  intercept) 

•  From:  high<risk.  accelerated  hardware  delivery  with  aggressive  DenWal  flight  test 
schedule  with  Low-Rate  Initial  Production  (LRIP)  21  months  after  EMD  Authority  to 
IVocecd  (ATP) 

•  To:  risk  averse  event-driven  flight  test  after  EMD  ATP  schedule  with  LRIP  60  montlu;  after 
EMD  ATP 


This  slide  shows  how  the  THAAD  program  schedule  has  slipped  because 
of  the  failed  flight  tests  and  subsequent  budget  cuts.  The  schedule,  as  outlined 
in  June  1994,  called  for  FUE  in  2002.  The  new  schedule,  shown  in  the  top  of 
the  figure,  adjusts  to  the  failed  flight  tests  and  concurrent  budget  cuts.  The 
completion  of  the  test  program  has  slipped  2  years,  and  FUE  has  slipped  from 
2002  to  2006. 

Even  now,  the  remaining  test  program  is  compressed,  and  the  criteria  for 
buying  the  40  UOES  missiles  remains  a  single  successful  test.  Consequently, 
the  early  capability  pressure  continues  to  compromise  the  test  missile 
instrumentation. 
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THAAD 

Aggressive  Test  Program 
Changing  Configurations  and  Goals 

•  The  test  conditions  changed  from  flight  to  flight 

~  Missile  configuration 

-  Guidance  software 

-  Intercept  conditions  (e.g.,  exo  vs.  high  endo) 

•  FT-8  wiD  fly  a  new  seeker  (InSb) 

•  Even  after  review  groups  recommended  not  changing  configurations,  THAAD 
wDl  fly  one  interceptor  on  ttie  upcoming  FT-8  and  FT-9  flights  and  a  different 
interceptor  on  the  FT-10  through  FT-13  flights 


The  test  missiles  in  the  THAAD  flight  test  program  have  undergone 
numerous  changes  between  between  FT-1  and  FT-7.  The  study  group  found 
that  the  guidance  software  was  written  only  to  work  on  that  particular  test  and 
under  the  expected  conditions.  Likewise,  hardware  changes  were  also 
occurring. 

The  new  InSb  seeker  will  fly  on  the  upcoming  FT-8.  This  will  be  the  first 
test  flight  for  this  seeker. 
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Sk>ine  THAAD  Project  Office  Concerns 
Regarding  the  THAAD  Contractor,  August  1996 

•  Fundamental  concerns  regarded  leadership  and  management 

•  Basic  philosophy  change  was  needed.  Must  adopt  a  system  perspective 

•  Lack  of  high-quality  personnel/functional  experts 

•  No  defined  system  engineering  process 

•  Little  evidence  of  system  engineering  talent  infusion 

•  Software  management  and  development  process 

-  Inadequate  system  level  management 

-  Disciplined  process  was  not  in  place 

-  Inadequate  requirements  documentation  and  stability 

•  Product  assurance  program  was  inadequate 


The  THAAD  program  office  also  expressed  concerns  with  the  contractor 
program  management  Again,  the  root  causes  were  associated  not  only  with 
the  technological  challenge  but  also  with  the  basic  set  of  disciplines  essential 
to  success  in  developing  and  testing  complex  systems. 

It  is  not  clear  what  actions  were  taken  as  a  result  of  these  concerns. 
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BMDO’s  Missile  Assessment  Team 
THAAD  Risk  Areas — ^April  1997 

•  Missile  design  moderate  risk  areas:  exo  and  high  endo 

-  Mission-critical  functions  -  System  margins 

-  Processor  usage  -  Seeker  design 

-  Autopilot  boost/TVC  -  Hit  point  performance 

“  Booster  separation  BSD  -  Wiring  and  grounding  design 

-  Instrumentation  design 

•  Design  evaluation/execution:  moderate  risk 

-  Missile  design  evaluation  -  IMUFCT 

-  lAP  FCT  -  Seeker 

-  One  shot  devices  -  Flight  simulation 

-  KV-level  ESS  -  KV  wiring 

-  DACSrcTandESS 

•  Design  evaluation/execution:  high  risk 

~  DACS  built-in  test  and  assembly  test 

-  Quality  assurance  program 


In  April  1997,  the  Missile  Assessment  Team  identified  specific  areas  of 
moderate-to-high  risk.  This  report  was  produced  after  the  failure  of  FT-7. 

This  list  of  significant  risk  areas,  provided  halfway  through  the  planned  test 
program,  should  have  been  compelling  evidence  of  the  compromising 
pressures  on  this  program.  Nonetheless,  in  late  1997,  the  program  direction 
was  still  driving  to  maintain  the  flight  test  schedule  to  meet  the  users’  “urgent” 
need. 
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THAAD  Risk  Assessment  and  Mitigation 
Areas  of  Moderate  and  High  Risk 
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The  evidence  is  that  the  continuing  incidence  of  moderate-  and  high-risk 
areas  this  late  in  the  test  program  is  not  caused  by  a  lack  of  understanding  of 
risk  areas.  The  THAAD  Program  has  spent  $700M  toward  risk  mitigation.  It 
is  more  likely  that  the  difficulty  lies  in  the  initial  unrealistic  schedule  and  the 
continued  attempts  to  keep  a  compressed  flight  test  schedule  moving  forward. 
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Remaining  DemA^al  Flight  Test  Program 
13  Flight  Program  for  THAAD 


Current  Remaining  PD&RR  Flight  Test  Schedule 


...  "i  V .  -  ..«v  1 

*a  *9 

#10  #11#12«13 

0  0 

0  ❖  ^  0 

■blihh 

_  1 

This  slide  shows  the  flight  test  schedule  as  of  early  February  1998.  FT-8 
has  now  been  delayed  because  of  difficulties  discovered  during  ground  testing. 
Even  so,  given  the  history  to  this  point,  the  remaining  schedule  still  appears  to 
be  compressed  and  optimistic,  particularly  since  the  end  of  the  test  period 
remains  unchanged  with  FT-1 1,  -12  and  -13  being  compressed  to  make  up  for 
the  slip  in  FT-8,  -9  and  -10. 
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THAAD  Analysis  Simulations 

The  Capabilities  Vs  The  Practice 

•  THAADSIM-HWIL 

~  Simlab  real-time  HWIL  simulation  (6  EXDF).  High-fidelity  simulation  for 
software  testing,  analysis,  and  qualification.  Also  hardware  integration  and 
interface  testing 

-  TOAADSIM-D 

-  Simlab  non-real-time,  all-digital  simulation  (6-DOF) 

•  THAADSIM-A 

-  Non-real-time,  all-digital  simulation  (6-DOF).  Reduced  fidelity  flight 
software  modules  to  allow  UOES,  EMD  analysis 

•  End-to-end 

-  THAAD  non-real-time,  all-digital  simulation  (S/6-DOF).  Analysis  of 
systems  level  sensitivities  to  target  uncertainties,  radar  errors,  in-flight 
updates,  and  so  forth 

-  Scientific,  engineering,  and  technical  assistance  (SETA)  sims 

-  SETA  contractor  simulations  (3  and  6  DOF).  Used  for  baseline 
comparisons  to  help  verify  models,  algorithms,  and  missile/weapons  system 
performance 


The  inadequate  simulation  and  ground  testing  is  probably  not  caused  by  the 
lack  of  facilities  or  capability.  The  study  group  found  the  combinations  of 
computer  simulation  and  HWIL  capabilities  to  be  impressive,  though  not 
complete. 

The  difficulty,  again,  seems  to  stem  more  from  a  mindset  that  limits  ground 
testing  to  the  minimum  essential  to  proceed  with  the  flight  test  schedule. 
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THAAD 

Observations 

•  HTK  is  a  very  difncult  task.  Attempting  an  operational  capability  before 
EMD  is  high  lisk  and  beyond  our  experience 

•  The  THAAD  program  places  a  high  priority  on  achieving  a  high-risk  DOES 
capability.  In  execution,  the  DOES  program  is  a  diversion  from  the  objective 
system 

•  The  DOES  requirement  was  designed  to  help  meet  an  urgent  need  but: 

-  Led  to  a  highly  concurrent  program 

-  Demanded  parallel  testing 

-  Compromised  the  missile  test  program 

-  Has  not  produced  early  capability 

-  Continues  to  impact  program  management  in  spite  of  rebaselining  and 
reevaluation 


The  study  group’s  purpose  was  not  to  evaluate  the  THAAD  program  but  to 
extract  lessons  that  would  be  useful  to  the  NMD  program.  Still,  immersion  in 
the  THAAD  program  led  to  the  obvious  observations  on  this  slide  and  the 
following  slide — observations  that  are  relevant  to  the  THAAD  program  and  to 
NMD  planning. 
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THAAD 

Observations  (Continued) 


•  Because  of  the  test  failures,  program  management  recognizes  the  need  for  a 
more  conservative  approach  with  performance  vs.  calendar  milestones. 
However,  schedule  pressures  persist 

•  Decision  to  exercise  40>missi]e  buy  is  still  based  on  a  single  successful 
intercept  vs.  the  more  conservative  LRIP  criteria  (3  intercepts) 

•  Need  clear  recognition  that  success  in  meeting  the  need — no  matter  how 
urgent — ^will  depend  on  a  design  with  adequate  margins  and  on  engineering 
and  fabrication  discipline 

•  The  program  has  been  examined  through  a  myriad  of  independent  reviews  that 
reported  these  problems 

-  Program  management  will  have  difHculty  adjusting  to  the  realities  of 
schedule  and  technical  chaUenge  as  long  as  the  intense  schedule  pressure 
persists 
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NTW  Theater  Ballistic  Missile  Defense  (TBMD) 

•  Purpose:  defend  against  medium-  to  long-range  TBMs  with  exo-almospheric 
intercepts  with  a  HTK  warhead 

•  Systems 

-  IBM  detection  and  tracking:  Defense  Support  Program  (DSP)  and 
AEGIS  system 

-  Missile:  Standard  Missile-3  (SM-3) 

•  HTK  with  LEAP 

-  C2:  AEGIS  Combat  System 


NTW  is  an  evolutionary  development  based  on  the  proven  AEGIS  and 
Standard  Missile  programs.  However,  it  adds  important  new  challenges:  SM-3 
flight  outside  the  atmosphere,  sensor  upgrades  to  meet  new  discrimination 
requirements,  and  a  variety  of  KV  challenges  with  the  LEAP  KV.  In  addition,  it 
includes  a  dual-bum  solid  rocket  third  stage  and  an  IR  thermal  guidance  system. 
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SM-3  Programmatic  Background 

•  Four  LEAP  intercept  attempts  (Anny  LEAP  2  and  3  and  Navy  Terrier  LEAP  3 
and  4) 

~  Four  different  configurations 

-  All  compressed  schedules 

-  All  failed 

•  Numerous  review  panels  and  ^lessons  learned**  studies.  Common  threads; 

-  Successful  intercept  demonstration  essential  to  continuation  of  the 
program 

-  Measured,  incremental  test  program  essential 

-  Configuration  variations  and  changes  must  be  minimized 

-  Comprehensive  end-to-end  system  testing  of  flight  configuration  essential 
to  risk  reduction 

-  Need  spare  missile  flight  assets 

-  Avoid  rush  to  failure 


As  is  the  case  with  THAAD,  operationally  useful  HTK  is  yet  to  be 
demonstrated  in  this  program  employing  different  HTK  technologies.  The  four 
attempts  in  the  precursor  Army  LEAP  and  Navy  Terrier  LEAP  program  were 
all  failures.  An  extensive  outside  review  by  an  independent  Blue  Ribbon  Panel 
led  to  the  decision  not  to  continue  the  Terrier  LEAP  program  but,  instead,  to 
move  to  the  AEGIS  Leap  Intercept  (ALI)  program  while  retaining  the 
emphasis  and  the  mandatory  milestone  of  demonstrating  HTK  capability. 

The  ALI  program  should  benefit  from  numerous  previous  reviews  that 
stressed  the  need  to  temper  the  drive  to  meet  the  “urgent”  need  with  the  reality 
of  the  difficulty  of  the  task. 


This  slide  illustrates  the  legacy  and  the  evolution  of  the  NTW  program.  As 
discussed,  the  ALI  program  follows  the  Navy’s  Terrier  LEAP  program  in  a 
continued  effort  to  prove  the  HTK  capability  of  the  LEAP  vehicle. 

The  NTW  program  has  been  significantly  modified  since  the  study  group 
began  its  work,  with  an  earlier  UOES  concept  giving  way  (at  least  on  briefing 
charts)  to  a  block  concept. 

Block  I  capability  is  seen  as  a  legacy  of  the  Navy  Area  Defense  Program. 
The  Area  Defense  program  uses  a  different  missile  (the  SM-2  Blk  IVA  missile 
with  an  explosive  warhead)  and  the  AEGIS  Combat  System. 

The  ALI  program  is  a  risk-reduction  activity  within  the  NTW  Defense 
program.  It  is  designed  to  prove  the  exo-atmospheric  operation  of  SM-3  and 
the  LEAP  vehicle. 
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NTW 

The  Program  Environment 

•  Initially,  intense  time  pressure  for  an  early  capability.  Mismatch  with 
department's  funding  priorities 

•  Higher  risk  considered  justified  by  “urgent”  fielding  requirement 

•  Multiple  additional  external  factors  contributing  to  risk: 

-  Funding 

-  Defense  industrial  base 
~  Political  pressures 

-  Testrange  adequacy  and  availability 

-  Targets 

•  Initial  phase  focused  to  demonstrate  that  LEAP  can  hit  a  target  in  space 


Intense  time  pressure  is  also  a  legacy  for  the  NTW.  Earlier  direction  from 
the  Secretary  of  the  Navy  and  the  Chief  of  Naval  Operations  (CNO) 
emphasized  the  urgency  for  NTW  and,  therefore,  the  willingness  to  accept  risk. 
There  may  also  be  some  sense  of  competition  with  THAAD. 

This  time  pressure  and  a  number  of  other  factors — including  significant 
resource  instability  and  an  incomplete  set  of  requirements — continue  to 
contribute  to  risk. 
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ALI  Program  Scope 

An  Essential  but  Limited  Step 

•  AEGIS  weapons  system 

-  AN/SPY-IB/D  radar 

-  Command  and  decision 

-  Weapons  control  system 

-  Vertical  launch  system 

•  SM-3  LEAP  interceptor 

-  SM-2  61k  IV  propulsion  chain 

-  Third-Stage  Rocket  Motor  (TSRM) 

-  KV  fourth  stage 

•  Aires  target 


The  ALI  program  is  the  currently  defined  phase  of  NTW.  Its  goal  is  to 
reduce  the  risk  of  exo-atmospheric  intercepts  by  performing  early  testing.  It  is 
a  tightly  focused  project  to  demonstrate  that  the  SM-3  can  deliver  the  LEAP 
vehicle  into  the  needed  basket  and  that  the  LEAP  vehicle  can  hit  a  target  in 
space.  It  does  not  include  the  kind  of  performance  specifications  that  will 
define  the  follow-on  operational  system  or  will  serve  as  the  basis  for  the  flight 
test  program.  That  work  is  yet  to  come. 


Top-Level  ALI  Missile  Requirements 

•  Threat  range:  500-1 ,000  km 

•  Missile  architecture  derived  from  SM-2  Blk  IV  and  Terrier  LEAP  hardware 
and  software 

•  V|„~3.4km/scc 

•  Nuclear  environment  not  imposed 

•  Lethality  and  discrimination  not  specified 


As  discussed,  many  of  the  essential  operational  characteristics  for  the  NTW 
are  not  yet  defined.  Hence,  while  success  in  the  ALI  program  is  an  essential 
precursor  to  NTW  development,  it  is  only  a  precursor  and  should  not  lead  to 
schedule  optimism  in  developing  and  testing  an  operational  system.  For 
example,  defining  and  meeting  discrimination  requirements  are  essential  for  an 
effective  operational  system.  This  could  be  at  least  as  daunting  as 
demonstrating  HTK  capability. 

The  NTW  program  has  realized  the  difficulty  of  solving  the  discrimination 
problem.  In  a  set  of  risk-reduction  activities  in  parallel  with  ALI  execution,  it 
has  funded  a  joint  system  level  [radio  frequency  (RF)/IR)]  discrimination 
effort  with  technical  support  from  the  Johns  Hopkins  University/Applied 
Physics  Laboratory  (JHU/APL);  the  Massachusetts  Institute  of  Technology/ 
Lawrence  Livermore  (MIT/LL);  the  Lockheed  Martin/Govemment  Electronic 
Systems  (LM/GES)  (AEGIS  Weapon  System);  and  the  Standard  Missile 
Company  (SMCo)  (SM-3  missile).  This  effort  is  to  determine  and  develop  the 
required  discrimination  algorithms  and  accompanying  equipment. 
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Overall  NTW  Program  Schedule 
(As  of  Fall  1997) 

Milestones 

21  J 

rV-4  (SPVcoea 

TBMO 

CapabUty 

A  A  1 

Art!  UOES  Area  Tactical  MTW  Blk  1 

Test  Flights 

aaa  a  aaaaaa 

11A  2  1  2346  67 

CTVi  FTHi 

The  flight  test  schedule  calls  for  seven  test  flights  over  a  period  of  4+ 
years.  This  schedule  is  well  within  the  realm  of  reason;  however,  given  the 
failure  of  CTV-1,  the  start  date  may  be  optimistic. 
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NTW  Test  Objectives  and  Schedule 
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At  the  moment,  it  appears  that  given  the  right  focus  on  qualifying 
components,  ground  tests,  and  preflight  review,  the  flight  test  schedule  and 
objective  now  appear  to  be  incremental  and  sequential. 
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This  slide  further  illustrates  the  incremental  and  sequential  nature  of  the 
currently  planned  flight  test  program.  The  study  group  was  unable  to  discover 
an  equally  well-defined  program  of  ground  testing. 
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SM-3  Integrated  Performance  Simulations 
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There  does  appear  to  be  a  reasonably  comprehensive  set  of  planned 
simulation  and  HWIL  test  capabilities  that  could  accommodate  a 
comprehensive  ground  test  program.  Some  combination  of  computer  and 
HWIL  is  available  to  evaluate  capability  from  launch  to  kill.  However,  it  is 
not  clear  that  these  simulations  are  connected  as  needed  to  provide  an  end-to- 
end  prefiight  evaluation. 
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ALI  Program’s  Risk  Items  for  SM-3 
December  1997 

•  High 

-  TSRM  forward  closure 

-  KW  envelope 
--  KW  weight 

~  KW  seeker  contamination 

•  Moderate 

-  Nose  cone  -  thermal  adequacy 

-  KWTE  console 

-  Nose  cone  structural  adequacy 

-  Nose  cone  separation  adequacy 

-  XSMDC  performance 

-  Concurrent  development  testing 
“  TSRM  case:  aero  heating 

-  Solid  Divert  and  Attitude  Control  System  (SDACS):  schedule 

-  TSRM  weight 

-  CTV>2LR1P  hardware  availability 


The  ALI  program  continued  to  carry  a  signifleant  list  of  high-  and 
moderate-risk  items  in  December  1997.  At  the  minimum,  this  suggests  that 
much  work  needs  be  done  to  conduct  a  flight  test  program  with  acceptable  risk. 
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study  Group’s  Observations  on  NTW 


•  The  program  was  initially  under  intense  time  pressure.  It  is  not  clear  bow  that 
has  been  adjusted  even  though  the  schedule  has  been  substantially  extended 

•  The  NTW  early  capability  goal  seems  on  the  path  to  the  objective  system,  but 
the  objective  system  is  only  partially  defmed 

•  The  ALL  program  is  sharply  focused  on  job  #1:  demonstrating  that  the  LEAP 
can  bit  a  target 

•  The  overall  ground  test  program  to  include  facility  use  was  not  yet  available. 

In  particular,  a  plan  has  to  be  developed  for  the  best  available  IfWIL 
simulation  of  the  KKV  seeker 

•  The  required  discrimination  capability — a  key  to  a  useful  operational  system — 
is  not  deHned 


This  slide  lists  some  of  the  more  obvious  observations  about  the  NTW 
program.  While  there  has  been  intense  time  pressure  to  field  some  early  NTW 
capability,  it  is  not  clear  whether  this  is  a  distraction  on  the  path  to  the 
objective  capability  since  that  capability  has  not  been  defined.  In  any  case,  the 
program  is  firmly  focused  on  the  essential  precursor  of  demonstrating  the 
ability  to  hit  an  incoming  target. 

While  there  is  a  reasonably  detailed  flight  test  program  for  the  ALI,  the 
study  group  was  unable  to  discover  an  adequately  detailed  ground  test 
program. 

Further,  a  key  operational  parameter  for  any  exo-atmospheric  system — ^the 
discrimination  requirement — ^is  yet  to  be  defined. 
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Patriot  Missile  Program 

•  Purpose:  point  defense  against  TBMs  using  low^ndo  intercept  with  a  HTK 
warhead  and  defense  against  cruise  missiles  and  attack  aircraft 

•  PAC-3  system  evolved  from  PAC-2 

-  ERINT  HTK  missile 

-  PAC-2  Guidance  Enhancement  Missile  (GEM)  missile 

-  Engagement  control  station 


The  Patriot  PAC-3  program  has  several  important  characteristics  that  are 
significantly  different  from  the  other  HTK  TMD  programs.  It  employs 
somewhat  more  mature  technologies.  More  importantly,  the  Patriot  program 
offices — ^government  and  contractor — ^are  mature  organizations  with  well- 
established  processes  and  a  proven  history  of  program  success.  Further,  the 
program  has  benefited  from  a  consistent  high  priority.  Hence,  while  HTK  is 
still  a  difficult  task  for  Patriot  PAC-3,  the  program  seems  well  organized  to 
deal  with  the  risks. 
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Patriot 

The  Program  Environment 

•  Acquisition  streamlining 

•  Budget  and  schedule  pressures 

•  Minimal  flight  test  and  test  resources 

-  2  Control  Test  Right  (CTF)  missiles 

-  16  PAC-3  Guided  Test  Right  (GTF)  missiles  (10  fired  at  TBM  targets) 

-  6  Patriot  missiles  (1  fired  at  a  TBM  target) 

•  Strong  reliance  on  simulations 

•  Roughly  4  years  from  the  start  of  contract  to  first  test  flight 


The  Patriot  program  has  a  full  plate  of  challenges.  Acquisition 
streamlining  impacts  the  way  the  government  and  the  contractor  share 
responsibilities.  While  the  impacts  in  the  long  term  are  likely  to  be  positive, 
the  short  term  may  be  turbulent 

Even  though  the  program  has  been  characterized  by  a  constant  high 
priority,  budget  and  schedule  pressures  still  abound.  One  consequence  of  these 
pressures  is  a  minimal  flight  test  program  with  strong  reliance  on  simulations. 
The  extensive  use  of  simulations  in  preparation  for  flight  test  should  be  a 
substantial  benefit  At  the  same  time,  the  cost-effectiveness  benefit  of 
simulations  in  lieu  of  an  adequate  flight  test  program  is  far  more  problematical. 

The  study  group  notes  that  the  PAC-3  missile  program  took  roughly 
4  years  from  the  start  of  the  contract  until  the  first  flight  This  program  built 
upon  a  base  of  contractor  experience  with  the  earlier  ERINT  system.  This  is  in 
contrast  to  the  THAAD  effort,  in  which  the  goal  was  first  flight  within  2  years 
of  the  contract  award. 
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Patriot  PAC-3  Program  Schedule 
As  of  November,  1997 
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Since  the  Patriot  system  has  a  dual  mission,  much  of  the  phased 
development  program  deals  with  radar  and  C2  requirements  for  the  air 
breathing  threat  and  for  the  TMD  threat. 

The  Quick  Response  Program  (QRP)  configuration  includes  radar 
enhancements,  remote  launch  capability  to  extend  the  missile  coverage  to  the 
limits  of  the  radar,  and  tactical  command  system  upgrades. 

Configuration  1  expands  the  C2  capabilities  and  introduces  the  PAC-2 
GEM. 

Configuration  2  includes  additional  enhancements  to  the  radar  and  C2 
system  and  some  survivability  enhancements. 

Configuration  3  includes  the  HTK  PAC-3  missile  along  with  additional 
radar,  remote  launch,  and  C2  development. 
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PAC-3  Missile  TBM  Flight  Test  Matrix 


MlMlon 

ObjMtIva 

DTI* 

Initial  flyout,  aaio  data,  aMramo,  uplnk/downlnk  (UL/Dt)  operation 

DT2* 

Flyout  to  extended  range.  Inflight  aignment,  themial  effects 

DT3 

MlssUe^System  Integration,  low  attitude 

DT4 

MtesHe-System  Integration.  hIgh-allKude 

DT5 

Low  magnitude  helix  maneuver,  remote  launch 

DT6a 

Accuracy  maneuver,  mullple  objects  In  seeker  field  of  view  (FOV) 

DT6a4) 

High-magnllude  helix  numeuver.  multiple  TBM,  turndown 

DT9a 

High-altilude  Intercept 

OTIa 

Multiple  threat  engagement 

OT2a> 

Low  radar  cross^secUon  (RCS)/hlgh-veloclly  target,  shoot-shoot  firing 

OT4a.b 

Mult^  TBM  engagement.  Inner  boundary  engagement 

*  Flights  DTI  and  DT2  did  not  involve  targett  or  intercept  irttempts 


This  slide  shows  that  part  of  the  flight  test  program  was  directed  at  the 
TBM  mission.  The  sequential  nature  of  objectives  is  noteworthy.  However,  it 
is  also  noteworthy  that  each  flight  breaks  new  ground  with  a  significant 
advances  in  functionality. 
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PAC  Models  and  Simulations 

•  PAC-3  SIM:  digital  end-to-end  system  simulation 

-  High-fidelity  radar,  6-DOF  missile,  missile  lethality 

•  Flight  mission  simulate  (FMS):  exercises  fully  integrated  ground  system 

-  HWIL  tactical  radar,  simulated  target  RF,  PAC-2  and  PAC-3  environments 

•  Guidance  Test  and  Simulation  Facility  (GTFS):  design  tool  -  support 

-  HWIL  ground  and  missile  pre-launch,  launch,  and  guidance  functions 

•  Missile  Command  (MICOM)  MSS-2:  launch-to-intercept  closed-loop  simulation 
to  measure  overall  system  performance 

-  HWIL:  ground  system  models  with  tactical  seeker,  3-axis  motion  table  in  an 
anechoic  chamber,  flight  dynamics  in  the  RF  environment 

-  Lockheed  Martin  Vought  Systems  (LMVS)  HWIL  Facility:  PAC-3  missile 
flight  hardware  and  software  integration  and  checkout 

•  Multifunction  simulation  (MFSIM):  high-fidelity  modeling  of  Patriot  system 
TBM,  non-TBM,  track,  guidance,  and  search  radar  loading 


As  mentioned,  there  is  an  extensive  set  of  simulations,  including  HWIL, 
available  for  the  Patriot  PAC-3  system. 
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DT*1  Flight  Test  Readiness  Reviews 
CTF  Missile 


•  PFTR  1.1:  completed  3  June  and  1  July  1997  in  Dallas,  Texas 

-  Assess  readiness  to  ship  DT-1  missile  forebody 

•  PFTR  1 .2  -  completed  18  September  1997  in  Huntsville,  Alabama 

—  Assess  integrated  system  readiness  for  final  DT-1  processing 

•  PFTR  1 .3  -  completed  25  September  1997  in  White  Sands  Missile  Range, 
New  Mexico 

-  Assess  Patriot  PAC-3  system  readiness  to  proceed  to  DT-1  launch 
DT-1  Test  Results:  Missile  System  Performance  Nominal 


The  Patriot  program  also  makes  use  of  an  extensive  flight  test  readiness 
review  process.  This  slide  shows  the  series  of  reviews  conducted  for  the  DT-1 
flight  of  fall  1997. 
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PAC-3  -  Observations 

•  While  the  program  builds  on  a  long  development  history,  HTK  is  still  a  new 
and  challenging  task 

-  The  PAC-3  missile  is  a  large  step  up  from  ERINT 

•  While  the  need  for  the  TMD  capability  is  urgent  and  PAC-3  is  a  high-priority 
solution  to  part  of  the  need,  the  program  urgency  does  not  seem  to  be  driving 
to  high-risk  approaches 

•  The  Patriot  PAC-3  program  seems  to  be  proceeding  toward  the  objective 
system  without  pressures  for  an  interim  capability 

•  The  long  history  of  the  Patriot  program  provides  a  legacy  of  disciplined  design 
and  engineering  processes 

•  The  program  makes  heavy  use  of  high-fidelity  simulations  and  HWIL 
simulations  to  reduce  flight  test  risk 

•  The  flight  test  program  seems  minimal,  compressed,  and  concurrent  The 
flight  test  program  and  key  program  milestones  are  compressed 


Some  highly  evident  observations  about  the  Patriot  PAC-3  program  are 
provided  in  this  slide. 
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The  NMD  Program 


In  this  section,  the  study  group  presents  some  relevant  information  about 
NMD  and  our  reactions  to  it. 


NMD  Program 

•  Purpose:  “to  develop,  demonstrate,  and  deploy  an  initial  land-based  national 
missile  defense  (NMD)  system  to  defend  against  limited  strategic  missile 
attacks  and  be  capable  of  evolving  to  counter  future  threats.** 

-  U.S.  defense  (all  50  states) 

-  Assured  human-in-control 

•  3  +  3 

-  3  years  (1999)  to  reach  readiness  to  deploy  in  3  years  (2002).  In  the 
interim,  continue  to  develop  and  improve  the  NMD  system 

-  Readiness  to  deploy  review  in  2000 

•  System  architecture:  “plug  and  defend** 

Flexible  systems  engineering  and  integration 

-  Respond  to  unknowns  in  threat  and  constraints 


The  current  approach  to  preparing  for  NMD  deployment  introduces  new 
and  very  demanding  complexities.  The  current  plan  calls  for  bringing  the 
system  to  a  state  of  maturity  by  the  end  of  1999  such  that  the  system  is  within 
3  years  of  deplo5m(ient.  In  the  worst  case,  that  would  require  deployment  in 
5  years. 

While  the  readiness  review  in  2000  would  theoretically  modify  the 
expectation,  the  very  high  visibility  of  this  program  is  likely  to  produce  intense 
pressures  to  maintain  the  schedule.  Hence,  the  program  begins  with  many  of 
the  problems  that  have  beset  other  HTK  programs:  compressed  schedule, 
minimal  flight  test,  and  poorly  defined  test  objectives. 

In  addition,  the  funding  for  the  program  has  been  erratic  at  best  and  the 
program  management  approach  depends  to  an  unprecedented  degree  on  an 
integrating  contractor— again,  an  intriguing  but  unproven  approach  for  a 
program  that  is  to  do  something  that  has  never  been  done  before. 
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NMD  Program  Structure 
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•  Deployment  Options  and  Upgrades 

•  Risk  Reduction 

•  Develop  and  Demonstrate 


^Flight  Test  ^^integrated  System  Test  (Right) 


IFT 1A/2  -  Sensor  Fly-by 

IFT  3/4  -  Intercept 

IFT  5  -  Integrated  System  Test 


•  Complete  Element  Development 

•  Integrate  Elements 

•  Demonstrate  kiitlal  System  Capability 


This  rendition  of  the  program  structure  calls  for  an  integrated  systems  test 
in  FY99  on  the  fifth  test  flight — about  1  year  after  the  first  intercept  attempt, 
which  occurs  on  the  third  test  flight 

There  is  one  backup  integrated  systems  test  flight  available  before  the . 
readiness  review  in  2000. 

This  schedule  appears  to  be  significantly  more  compressed  and  optimistic 
that  the  TMD  programs  schedules  that  have  proven  to  be  excessively 
challenging. 
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NMD  System  Risks 


System  Risk 

Concern 

Mitigation 

T4E  Program 

Scopg 

•  Number  of  tests  before 
deployment 

•  Increased  funding /added 
more  tests  /  added  spares 

Systsm 

Discrimination 

•  AbVity  of  elements  to  acquire 
data,  fuse  data,  and 
discrtminate  reentry  vehicle  (RV) 
in  a  tlme>oonstralned  environment 

•  Increased  development, 
testing  and  validation  of 
discrimination  algorithms 

•  Dual  EKV  sensor  approach 

System  Siting 

•  Construction  tlmsNnes 

•  Easements 

•  Environmental  Impact  statements 

•  Incremental  deployment 

•  Congressional  waivers 

•  Selected  sKes/EIS  pfxx:ess 

External 

integration 

•  Stressing  timeline  of 
required  Integration  with 
external  agencies 

•  Integration  plans  developed 
and  being  implemented 

The  program  office  recognizes  the  fact  of  high  risk  associated  with  the 
minimal  flight  test  schedule  and  the  very  difficult  discrimination  challenge. 
Non-technical  program  challenges  are  also  recognized.  However,  the  program 
has  not  yet  matured  to  the  point  of  identifying  the  specific  set  of  risks  that  need 
to  be  addressed.  That  activity  will  have  to  await  the  work  of  the  integrating 
contractor. 


67 


NMD  Test  Risk 
Test  and  Target  Spares 

•  Spare  booster  available  for  all  flights 

•  Front  section  spares  fonned  from  a  combination  of  spare  components  and 
pulling  subsystems  from  succeeding  planned  flights 

•  Target  spares  formed  by  pulling  target  object  from  succeeding  planned  flights 


Spare  test  assets  are  a  partial  answer  to  the  high-risk  test  program. 
However,  it  appears  that  the  current  plan  for  obtaining  spares,  shown  on  the 
slide,  is  minimal  at  best. 
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NMD  Risks  and  Mitigation 
System  Discrimination  (EKV) 

•  Risks 

-  Battle  Management/Command,  Control,  and  Communications  (BM/C3) 
fusing  different  types  of  sensor  data 

-  EKV  combining  IR  sensor  images  with  ground-based  interceptor  (GBI)  IR 
discrimination  algorithms 

-  EKV  target  discrimination 

•  Mitigation 

-  Parallel  EKV  contractor  competition 

-  Transition  technology  activities 

•  Rad  hardening/meicad  telluride  technology 

•  Focal  plane/silicon  array  technology 


The  discrimination  problem  for  NMD  is  very  challenging.  The  information 
on  this  slide  was  provided  by  the  NMD  JPO.  The  mitigation  techniques 
proposed  here  are  noteworthy  but  are  not  adequate  to  cover  the  range  of  risks 
already  identified. 
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NMD  T&E  Risk  Reduction 

Risk  Areas  far  T^E  Resolution 

BMDO^Proposed  T&E  Risk-Reduction  Actions 

•  Limited  system  level  testing 

•  Additional  integration  facility  forpie-IFT  test 

•  Threat  target  realism 

•  Target  object  inventory:  RVs,  decoys,  balloons,  and 

•  Lack  of  spare  test  articles 

so  forth 

•  Multiple  target  tests 

•  Spare  test  expendables 

•  NMD  interoperability  testing 

•  Upgrade  launch  support  at  Meek  Island 

•  Increased  risk-reduction  flight  tests 

•  More  simulation,  test,  and  evaluation  process 

Specific  T&E  risks  are  currently  being  addressed  by  the  NMD  JPO  with 
this  set  of  risk-mitigation  measures.  While  these  measures  seem  necessary, 
they  do  not  do  much  to  relieve  a  very  demanding  development  and  test 
schedule. 
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Meetings  and  Site  Visits 


Meetings  of  the  Study  Group  at  the  Institute  for  Defense  Analyses  (IDA) 

•  September  25-26, 1997 

•  November  5-6. 1997 

•  December  12, 1997 

•  January  15-16. 1998 

Site  visits  occurring  in  December  1997  and  January  1998 

•  Kinetic  Kill  Vehicle  Hardware-in-the-Loop  Simulation  (KHILS)  Facility,  Eglin 
Air  Force  Base  (AFB),  Florida 

•  U.S.  Army  Aviation  and  Missile  Command  (AMCOM),  Huntsville,  Alabama 

•  Standard  Missile  Company  (SMCd),  Tucson,  Arizona 

•  Lockheed  Martin,  Sunnyvale,  California 

•  Lockheed  Martin  Vought  Systems  (LMVS),  Arlington.  Texas 

•  Pacific  Missile  Range  Facility  (PMRF).  the  island  of  Kauai  in  Hawaii 

•  Integrated  System  Test  Capability  Facility,  Huntsville,  Alabama 

•  Guidance  System  Evaluation  Laboratory,  Laurel,  Maryland 
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GLOSSARY 

AFB 

Air  Force  Base 

ALI 

AEGIS  LEAP  Interceptor 

ATP 

Authority  to  Proceed 

BM/C3 

battle  management/command,  control,  and 
communications 

BM/C3I 

battlemanagement/command,  control, 
communications,andintelligence 

BM/C4I 

battlemanagement/command,  control, 
communications,  computers,  and  intelligence 

BMD 

ballistic  missiledefense 

BMDO 

Ballistic  Missile  Defense  Organization 

C2 

command  and  control 

CDP 

Continuing  Development  Period 

CDR 

Critical  Design  Review 

CIL 

computer-in-the-loop 

CNO 

Chief  of  Naval  Operations 

COEA 

Cost  and  Operational  Effectiveness  Analysis 

CSEDS 

Combat  Systems  Engineering  Development  Site 

CTF 

Control  Test  Flight 

CTV 

Control  Test  Vehicle 

DAB 

Defense  Acquisition  Board 

DACS 

Divert  and  Attitude  Control  System 

dB 

decibel 

DDR&E 

Director  of  Defense  Research  and  Engineering 

DemA^al 

demonstration/validation 

DoD 

Department  of  Defense 

DOF 

degrees  of  freedom 
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DOT&E 

Director,  Operational  Test  and  Evaluation 

DRR 

Deployment  Readiness  Review 

DSP 

Defense  Support  Program 

DT 

development  test 

DT&E 

development,  test,  and  evaluation 

DTSE&E 

Director,  Test,  Systems  Engineering  and  Evaluation 

EIS 

Environmental  Impact  Statement 

EKV 

Exo-atmosphericKill  Vehicle 

EMD 

EngineeringManufacturingDevelopment 

EOC 

Early  Operational  Capability 

ERINT 

Extended  Range  Interceptor 

ERIS 

Exo-atmospheric  Reentry  Intercept  System 

ESS 

environmental  stress  screening 

FCT 

flight  confidence  test 

FDR 

Final  Design  Review 

FDRU 

Final  Design  Review  Update 

FMS 

flight  mission  simulator 

FOV 

field  of  view 

FPA 

focal  plane  array 

FRP 

full  rate  production 

FRR 

Flight  Readiness  Review 

FTR 

Flight  Test  Round 

FTV 

Flight  Test  Vehicle 

FUE 

first  unit  equipped 

FY 

fiscal  year 

GBI 

ground-based  interceptor 

GBR 

ground-based  radar 

GEM 

Guidance  Enhancement  Missile 

GN&C 

guidance,  navigation,  and  control 

GIF 

Guided  Test  Flight 
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GTFS 

Guidance  Test  And  Simulation  Facility 

HOE 

Homing  Overlay  Experiment 

HTK 

hit-to-kill 

HWDL 

hardware-in-the-loop 

lAP 

Integrated  Avionics  Package 

ICBM 

Intercontinental  Ballistic  Missile 

IDA 

Institute  for  Defense  Analyses 

IDR 

Initial  Design  Review 

IFT 

Integrated  Flight  Test 

IMU 

inertial  measurement  unit 

InSb 

Indium  Antimonide 

IOC 

initial  operational  capability 

lOT&E 

initial  operational  test  and  evaluation 

IPT 

Integrated  Product  Team 

IR 

infrared 

rn> 

Integrated  Test  Plan 

JHU/APL 

Johns  Hopkins  University/Applied  Physics 
Laboratory 

IPO 

Joint  Project  Office 

KHILS 

Kinetic  Kill  Vehicle  Hardware-in-the-Loop 
Simulation 

KKV 

KineticKill  Vehicle 

km 

kilometer 

KV 

killvehicle 

KW 

kineticwarhead 

LEAP 

Lightweight  Exo- Atmospheric  Proj  ectile 

LL 

LongLead 

LLI 

Long  Lead  Item 

LM/GES 

Lockheed  Martin/Govemment  Electronic  Systems 

LMVS 

Lockheed  Martin  Vought  Systems 

LRIP 

Low  Rate  Inidal  Production 
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MCO 

missile  checkout 

MFSIM 

multifunction  simulation 

MICOM 

Nfissile  Command 

MICOM/SED 

Mi  ssile  Command/Software  Engineering  Directorate 

MIT/LL 

Massachusetts  Institute  of  Technology/Lawrence 
Livermore 

MSI 

hfilestone  I 

Msn 

Milestone  n 

Msm 

Milestone  III 

NDT 

non-destructive  test 

NMD 

National  Missile  Defense 

NTW 

Navy  Theater  Wide 

DIPT 

Overarching  Integrated  Product  Team 

PAC 

Patriot  Advanced  Capability 

PD&RR 

Program  Definition  and  Risk  Reduction 

PDR 

Program  Design  Review 

PEELS 

Patriot  End-to-End  Lethality  Simulation 

PFTR 

Patriot  Flight  Test  Review 

PLS 

Palletized  Load  System 

PMRF 

Pacific  Missile  Range  Facility 

PtSi 

PlatinumSilicide 

QDR 

Quadrennial  Defense  Review 

QRP 

Quick  Response  Program 

R&D 

research  and  development 

RCS 

radar  cross-section 

RF 

radio  frequency 

SDACS 

Solid  Divert  and  Attitude  Control  System 

SDIO 

Strategic  Defense  Initiative  Organization 

SDR 

Systems  Design  Review 

SETA 

scientific,  engineering,  and  technical  assistance 

SM-2 

Standard  Missile-2 
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SM-3 

Standard  Missile-3 

SMCo 

Standard  Missile  Company 

SRR 

System  Requirements  Review 

SWIL 

sofhvare-in-the-Ioop 

T&E 

teat  and  evaluation 

T/R 

transmit/receive 

TBMD 

theater  ballistic  missile  defense 

TEMP 

Test  and  Evaluation  Master  Plan 

THAAD 

Theater  High  Altitude  Area  Defense 

TMD 

theater  missile  defense 

TPA 

test  planning  analysis 

TSRM 

Third-Stage  Rocket  Motor 

TVC 

Thrust  Vector  Control 

UL/DL 

uplink/downlink 

UOES 

User  Operational  Evaluation  System 

USD(A&T) 

Under  Secretary  of  Defense  for  Acquisition  and 
Technology 

V&V 

validation  and  verificati  on 

Vbo 

velocity  at  burnout 
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